Musical Mosaicing with High Level Descriptors

This thesis investigates the use of high level descriptors (like genre, mood, instrumentation, singer's gender, etc.) in audio mosaicing, a form of data driven concatenative sound synthesis (CSS). The document begins by discussing the advances made in the eld of music content description over the last 10 years, explaining the meaning of high level music content description and highlighting the relevance of automatic music content description in general, to the eld of audio mosaicing. It proceeds, tracing the origins of mosaicing from its beginnings as a time consuming manual process, through to modern e orts to automate mosaicing and enhance the productivity of artists seeking to create mosaics. The essential components of a mosaicing system are described. Existing mosaicing systems are dissected and categorised into a taxonomy based on their potential application area. The time resolution of high level descriptors is investigated and a new hierarchical framework for incorporating high level descriptors into mosaicing applications is introduced and evaluated. This framework is written in Python and utilises pure data as both user interface and audio engine. Descriptors, stemming from Music Information Retrieval (MIR) research are calculated using an in-house analysis extraction tool. In-house audio-matching software is used as the similarity search engine. Many other libraries have also been integrated to aid the research, in particular Aubio for note detection, and Rubberband, for time stretching. The high level descriptors included in this project are; mood (happy, sad, relaxed or happy), gender (male or female), key, scale (major or minor), instrumental, vocal. A mini application for augmenting audio loops with mosaics is presented. This is used to show how the framework can be extended to cater for a given mosaicing paradigm. The musical applications of mosaics in the traditional song-based composition are also explored. Finally, conclusions are drawn and directions for future work postulated.